Providing A Collaborative Immersive Environment Using A Spherical Camera and Motion Capture

ABSTRACT

Embodiments of the present invention provide a collaborative visualization system, which integrates motion capture and virtual reality, along with kinematics and computer-aided design (CAD), for the purpose of, for example, validating a simulation with a real-world video. A spherical camera captures real-world video at a first location. At a second location, one or more head-mounted display devices display the captured real-world video and also display a simulation corresponding to the real-world video. A motion capture system captures head rotation information for one or more users to thereby to control a pan, tilt, and zoom of the real-world video so that when a position of a user&#39;s head changes, the portion of the real-world video displayed in the head-mounted display changes accordingly. Upon user input, a computer program product selects between displaying the real-world video and the simulation in the head-mounted display.

RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application Ser. No. 61/022,185, by Dobbins et al., titled“System and Program Product for Providing a Collaborative ImmersiveEnvironment and Related Methods” filed Jan. 18, 2008, incorporatedherein by reference in its entirety.

BACKGROUND

1. Field of Invention

The present invention relates generally to virtual reality and motioncapture, and, more particularly, to systems and program products whichallow persons to interact with real and artificial environments usingmotion capture.

2. Background

Various techniques and technologies exist which allow users to interactwith or analyze their environment. For example, motion capturetechniques are used in the fields of sports, medicine, andentertainment, especially video gaming and animation. In sports, forexample, motion capture enables a golfer's swing to be digitallyrecorded for analysis. In medicine, orthopedic rehabilitation can employmotion capture to provide feedback to the patient, illustrating corrector incorrect techniques with the patient's movements during walking, forexample. In animation, motion capture allows for an actor's movementsand even facial expressions to be digitally recorded in a computermodel. Later, animators use the actor's recorded motions as the basisfor the motions of a computer-generated character. Likewise, video gamesuse motion capture to facilitate the animation of life-like characterswithin the games.

Virtual reality technologies allow a user to interact with acomputer-simulated environment. Most virtual reality environments relyon computer screens or stereoscopic displays and are primarily visualexperiences. A popular example of virtual reality technology is a flightsimulator video game, in which the player pilots a virtual aircraft in acomputer-simulated environment.

Telepresence refers to technologies which allow the user to experience,or be present at, a remote location. For example, telepresence includesa remote video camera in which the user can control the pan, tilt, andzoom, if the display is of sufficient size and quality to allow the userto feel present at the remote location.

None of these technologies alone provide a collaborative immersiveenvironment for evaluating a design through interaction, virtualtraining on a task, and validating a simulation with a life video.

SUMMARY OF INVENTION

In view of the foregoing, embodiments of the present invention, forexample, provide a collaborative visualization system, which integratesmotion capture and virtual reality, along with kinematics andcomputer-aided design (CAD), for the purpose of, amongst others,evaluating a design. Embodiments of the present invention also provide,e.g., portable motion capture systems, which allow one or more personsto interact with real and artificial environments, and head mounteddisplays for evaluating a design, virtual training on a task, andvalidating a simulation with a real-world video, amongst others.Embodiments of the present invention further provide, for example,objects to be tracked by a motion capture system and incorporated into avirtual reality simulation. Embodiments of the collaborativevisualization system can include, for example, an immersive environmentwith one or more simultaneous users, with one or more externalobservers, with real-time interactions and scaling, and with the abilityto switch between a simulation and a telepresence view. An immersiveenvironment can, for example, generate a three-dimensional orstereoscopic image which appears to surround the viewer. That is, theviewer is “immersed” in the artificial environment.

Embodiments of the present invention include, for example, a virtualreality simulator. The virtual reality simulator can receive data fromCAD designs and display a simulation constructed from the data to a uservia, for example, a head mounted display, resulting in full-scale andstereoscopic images. The images can be, for example, so detailed thatthe user can read the wording on the working knobs and switches withinthe simulation.

Embodiments of the present invention further include, for example, amotion capture system incorporated into the virtual reality simulator.The motion capture system, for example, can track the movements andinteractions of a user (who is wearing a motion capture ensemble withflexible and adjustable level of detail, from just the head to a fullbody suit and gloves) within the virtual reality simulation so that thewhen the user's head rotates or tilts, the view of the simulationrendered in the head mounted display changes accordingly. The motioncapture system, for example, can store the tracked movements andinteractions of a user for later use, including, for example, trainingor design evaluation purposes. The motion capture system also can trackthe movements and interactions of multiple users within the virtualreality simulation such that the multiple users are represented byavatars in real time within the simulation. Thus, multiple users cansimulate a coordinated activity within the simulation, such asperforming routine maintenance on an aircraft, for example. In addition,the motion capture system, for example, can track the movements andinteractions of objects, including tools and props used by a user.

Embodiments of the present invention also include, for example, animmersive observation system where multiple observers can view, inreal-time, the virtual reality simulation including the interactions ofthe avatars in a common, immersive environment so as to evaluate the CADdesign. The immersive observation system can include a CAVE (CaveAutomatic Virtual Environment), such as, a reconfigurable 8-foot-high by10-foot-wide by 10-foot-long room with displays on three walls and thefloor, where one or more observers view a common environment in animmersive and interactive way, including stereoscopic and full-scaleimages. Advantageously, using the CAVE, the designers of the CAD designcan observe end-users interacting extensively with the proposed designin the simulator to analyze and evaluate the design without having tobuild expensive prototypes. For example, observers can view users withinthe simulation performing the routine maintenance operations on theaircraft. In addition, trainees can observe, for example, trainersperforming various tasks by viewing avatars of the trainers responsiveto recorded motion capture data.

According to the embodiments of the present invention, the virtualreality simulator can scale each avatar in real time. That is, a 5′ 4″user within the simulation can be scaled in real-time to be a 6′ 2″avatar within the simulation. The images rendered in the 5′ 4″ user'shead mounted display will correspond to the perspective expected bysomeone 6′ 2″. An observer of the simulation will see a 6′ 2″ avatar.Scaling may be accomplished by applying a ratio to each aspect of thedata in the translation from motion capture data to simulation data.Alternately, scaling may be accomplished in real time by positioning theavatar's head, hands, and feet in the correct location to allowkinematics software to solve for the other joints. In addition, scalingmay be accomplished in post processing, as opposed to in real time.

According to embodiments of the present invention, the virtual realitysimulator can include interactions with the simulated environment. Inone such example, the virtual reality simulator can include collisiondetection software to provide feedback within the simulation. If a usersticks his or her hand where the simulation indicates that a wall shouldbe, for example a virtual collision is detected. The hand motion is setto either stop on the collision or allowed to disappear from the view ofthe user (because it is, after all, behind the wall), and the panel ofthe wall can be set to change color (or some other behavior) to providefeedback and indicate that a collision has occurred. In a preferredconfiguration, the wall turns red; similarly, if a knee “collides” witha toolbox in the simulation, the toolbox turns red. In exemplaryconfiguration, the collision triggers a sound; the sound can be adirectional sound indicating a direction of the collision with respectto the user or observer. Various types of sounds can further provideinformation regarding the collision, such as, severity or the objectsinvolved in the collision. For example, a user hitting the user's headon part of an aircraft can result in a different sound than a objectcolliding with the floor. In addition, the simulation can alter itsbehavior based on detected collisions, by opening a door or panel, forexample.

In addition, embodiments of the present invention can include a portablemotion capture system for capturing tasks in the field. This systemincludes motion tracking markers (perhaps on a suit, body, or otherapparel), a plurality of cameras installed on a tripod or clamped on arigid structure so that cameras can track the movements of a userwearing the motion capture markers, and a computer to record the imagesfrom the camera. The portable motion capture system allows for a remoteprocedure, such as a field maintenance operation, be recorded. Becauseof the incorporated nature of the virtual reality simulator and themotion capture system provided by the embodiments of the presentinvention, the data from a field maintenance operation can later bestudied in the virtual reality simulator for interactions with a newdesign or for real-time evaluation of, for example, an existing designor a sequence of operations on an existing design. Moreover, accordingto embodiments of the present invention, a portable motion capturesystem can be utilized for real-time design evaluation in the field, forpresentation of a design, and for training at a remote location.Evaluation of a design can include, for example, evaluating a designwith respect to an environment, an analysis of the ergonomics of thedesign, a study of tasks associated with the design, and otherconsiderations as understood by those skilled in the art.

Furthermore, embodiments of the present invention include methods ofvalidating a simulation with real-world video using immersivetechnology. For example, according to an embodiment of such a method, aspherical camera captures real-world video, or a real-world stillphotograph, at a remote location. Later, the video or photograph isrendered in a head mounted display. A motion capture system collects theuser's head rotation information, which is used to control the pan,tilt, and zoom of the video. Then the user can switch between displayingthe real-world video and the simulation as a way of validating thesimulation. In addition, the video can be displayed on a desktop orCAVE. As an example, using a spherical camera to capture the real-worldimages from the deck of an aircraft carrier can be used to validate asimulation of that environment.

Embodiments of the present invention include, for example, systems andassociated methods of providing an immersive environment with multiplesimultaneous users, with external observers, with real-time interactionsand scaling, and with the ability to switch between a simulation and atelepresence view, as will be understood by those skilled in the art.Embodiments of the present invention provide improved approaches toevaluate designs without having to build prototypes and to trainpersonnel without the need for prototypes or on location travel.

BRIEF DESCRIPTION OF DRAWINGS

So that the manner in which the features and benefits of the invention,as well as others which will become apparent, may be understood in moredetail, a more particular description of the invention brieflysummarized above may be had by reference to the embodiments thereofwhich are illustrated in the appended drawings, which form a part ofthis specification. It is also to be noted, however, that the drawingsillustrate only various embodiments of the invention and are thereforenot to be considered limiting of the invention's scope as it may includeother effective embodiments as well.

FIG. 1 is a schematic diagram of a system to provide a collaborativeimmersive environment for the evaluation of an engineering designaccording to an embodiment of the present invention;

FIG. 2 is an environmental view illustrating four users wearing motioncapture equipment interacting with a virtual reality simulation,according to another embodiment of the present invention;

FIG. 3 is an environmental view of a motion capture glove according toan embodiment of the present invention;

FIG. 4 is a perspective view of a Hergo Easy Mount Mobile ComputerCart—Star Base according to an embodiment of the present invention;

FIG. 5 is a perspective view of a head-mounted display according to anembodiment of the present invention;

FIG. 6 is a perspective view of a head tracker to be used in conjunctionwith the CAVE according to an embodiment of the present invention;

FIG. 7 is an environmental view of wand hardware to be used inconjunction with the CAVE according to an embodiment of the presentinvention;

FIG. 8A is a perspective view of a spherical camera according to anembodiment of the present invention;

FIG. 8B is a perspective view of motion capture cameras according to anembodiment of the present invention;

FIG. 9 is an perspective view of avatars within a virtual realitysimulation according to an embodiment of the present invention;

FIG. 10 is a schematic block diagram of a collaborative visualizationsystem according to an embodiment of the present invention;

FIG. 11 is a schematic flow diagram of a method to provide acollaborative immersive environment for the evaluation of an engineeringdesign according to an embodiment of the present invention; and

FIG. 12 is a schematic flow diagram of a method of validating asimulation with real-world video using immersive technology according toan embodiment of the present invention.

DETAILED DESCRIPTION OF INVENTION

The present invention will now be described more fully hereinafter withreference to the accompanying drawings, which illustrate embodiments ofthe invention. This invention may, however, be embodied in manydifferent forms and should not be construed as limited to theillustrated embodiments set forth herein; rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of the invention to those skilled in the art.Like numbers refer to like elements throughout.

As illustrated in FIGS. 1 and 10, embodiments of the present inventioninclude an collaborative visualization system 20 which integrates motioncapture and virtual reality technologies, along with kinematics and CAD,for the purpose of, amongst others, evaluating a design, virtualtraining on a task, and validating a simulation with a real-world video.In addition, embodiments of the present invention include, for example,immersive observation environments as well.

Embodiments of the present invention include, for example, a virtualreality simulator 58 to create the virtual reality environment. Thevirtual reality simulator can receive data from a CAD program and createa virtual reality simulation of the design projected to a user via ahead mounted display 40, as illustrated, for example, in FIGS. 1, 2, and5. In addition, the simulation may be displayed via a desktop or a CAVE44. The virtual reality simulation is three-dimensional (3D) and allowsa user to inspect, evaluate, and interact with the CAD design. Thesimulation environment is labeled immersive because the simulation is 3Dand full-scale, and the user's view can rotate throughout the simulationso that and the user becomes immersed in the simulation.

Embodiments provide for evaluation of designs for aircraft, spacesystems, spacecraft, ships, and missile systems, which often utilize anextensive evaluation process traditionally requiring, for example,expensive mock-ups and prototypes. The extensive evaluation process canadvantageously include ergonomic analysis and task analysis foroperation and maintenance tasks, as understood by those skilled in theart.

According to an exemplary embodiment of the present invention, thevirtual reality software used includes ENVISION (D5) from DELMIA. Asunderstood by those skilled in the art, this software provides aphysics-based, 3D environment specifically for designing, verifying, andrapid prototyping of concept designs involving structures, mechanicalsystems, and humans. As understood by those skilled in the art, thesoftware enhances system and subsystem level models with physics-basedmotion, virtual reality immersion, and ergonomic evaluation capabilitiesfor highly accurate 3D simulation, analysis, and visualization. Inaddition, according to an exemplary embodiment of the present invention,other software components used to create the virtual reality environmentinclude: PolyWorks from InnovMetric Software Inc, a software tool usedto convert point cloud data to polygons; NuGraf from Okino ComputerGraphics, a polygon conversion and reduction tool; Deep Exploration orDeep Server from Right Hemisphere, a polygon conversion and reductiontool; and MS Visual Studio from Microsoft, a code development suite.According to an exemplary embodiment of the present invention, thehardware supporting this software can includes: four Dell PrecisionWorkstations 670, 16x DVD-ROM, 48/32 CDRW, Dual 3.0 Ghz Xeon with 2 MBL2 Cache, 800 FSB 4 GB RAM, nVidia Quadro PX3400 256 MB, 136 GB HD. Asunderstood by those skilled in the art, the virtual reality simulatorcan include a computer program product, stored in one or more tangiblecomputer readable media and readable by a computer so that the computerprogram product operates to perform the various instructions when readby the computer as described herein.

As illustrated in FIGS. 1, 2, and 3, according to an embodiment of thepresent invention, the motion capture system 30 includes, for example,users wearing bodysuits 32, gloves 34, and headgear 36 with markers 52at known locations on the suit, such as the knee, the wrist, and the topof the shoulders. The motion capture system can further include realobjects, or props, also having markers to be represented in the virtualreality simulator. Cameras 54, as illustrated in FIG. 8B, then digitallyrecord the locations of the markers as the users move around andinteract in the simulation, capturing a set of data. This motion capturedata can then made available, in real time, to the virtual realitysimulator 58 so that the users and their movements are modeled asavatars 56 within the simulation and so that feedback is provided to theusers. An avatar is an electronic image that represents and ismanipulated by, or driven by a computer user, as in a computer game orother virtual reality setting, typically being an incarnation in humanform.

According to an exemplary embodiment of the present invention, themotion capture system 30 includes up to twenty-four cameras 54 (e.g., 12Eagle-i and 12 Hawk-i from Motion Analysis Corporation, as shown in FIG.8B), for example, mounted on a truss 38 (e.g., L×W×H is 20′×5′×10′) usedto track up to six concurrent users wearing head-mounted displays 40,gloves 34 (e.g., TALON Gloves from Motion Analysis Corporation), andbody suits 32. According to an embodiment of the present invention, themotion capture system 30 uses software from Motion Analysis Corporationcalled EVART V5.0.4 and includes the following plug-ins: AnimationPlugins, RT2 Animation Plugins, Calcium 4, Talon Streaming 4, TalonViewer 4, and EVaRT5. As understood by those skilled in the art, thissoftware allows templates and props to be created and customized fortracking everything from physical mockups to full body person. Inaddition, VRSim's SimIO module can be used to multicast the datacollected by EVaRT to be used by the simulation engine software ENVISIOND5. As understood by those skilled in the art, embodiments can, forexample, incorporate any number of cameras.

According to an embodiment of the present invention, the virtual realitysimulator can, for example, scale each avatar in real time. That is, a5′ 4″ user within the simulation can be scaled in real-time to be a 6′2″ avatar within the simulation. The images rendered in the 5′ 4″ user'shead mounted display will correspond to the perspective expected bysomeone 6′ 2″. An observer of the simulation will see a 6′ 2″ avatar,and the avatar's posture will match that of the user. Scaling may beaccomplished by applying a ratio to each aspect of the data in thetranslation from motion capture data to simulation data. Alternately,scaling may be accomplished by positioning the avatar's head, hands, andfeet in the correct location to allow kinematics software to solve forthe other joints. FIG. 9 illustrates 4 avatars scaled to differentsizes, all driven from the same user; hence, all have the same posture.Note also that scaling can be accomplished in post processing, asopposed to in real time. That is, for a remote training example, a user,i.e., a trainee, of a first size, e.g, 6′ 2″, can be displayed an avatarof the first size, e.g, 6′ 2″, responsive to motion capture data from auser, i.e., a trainer, of a second size different than the first size,e.g, 5′ 4″.

As illustrated in FIGS. 1, 2, and 5, according to embodiments of thepresent invention, a user experiences the virtual reality environmentvisually through a head-mounted display 40, and each of the head mounteddisplays can have a different perspective of the virtual realitysimulation. The head-mounted display 40 can include a stereo displayhelmet worn by users for immersive visualization of full-scale data. Asunderstood by those skilled in the art, one type of head-mounted display40, the VR1280 from Virtual Research, has the ability to display at1280×1204 at 60 Hz in mono or stereo. As understood by those skilled inthe art, a head mounted displays can include separate left-eye andright-eye displays with different images so that a user views an imagein the head mounted display stereoscopically.

In an exemplary embodiment of the present invention, the software usedto display the 3D scene can be based on OpenSceneGraph 3D graphicstoolkit. MiniViz from VRSim is a viewer that allows the user to view arunning ENVISION (D5) simulation. The viewer loads models in theenvironment and then references the ENVISION (D5) simulation for thepositions of the models and tracked viewpoints.

Embodiments of the present invention provide computer workstations tosupport the head-mounted displays. In an exemplary embodiment of thepresent invention, the hardware used to support four head-mounteddisplays includes: four Dell Precision Workstation 670, 16x DVD-ROM48/32 CDRW, Dual 3.0 Ghz Xeon with 2 MB L2 Cache, 800 FSB, 4 GB RAM,nVidia Quadro FX3400 256 MB, 136 GB HD. For convenience and asunderstood by those skilled in the art, a Hergo Easy Mount MobileComputer Cart—Star Base 42, as illustrated in FIG. 4, can be used tomount each set of equipment, including the computer, keyboard, mouse,head-mounted display 40, Talon Gloves 34, and a flat panel monitor fromHergo, according an embodiment of the present invention.

According to embodiments of the present invention, the virtual realitysimulator can include, for example, interactions with the simulatedenvironment. In one such example, the virtual reality simulator caninclude collision detection software to provide feedback within thesimulation. If a user sticks a hand where the simulation indicates thata wall should be, a virtual collision is detected. The hand can bestopped at the time of the collision or be allowed to disappear from theview of the user and the panel of the wall change color (or some otherbehavior) to provide feedback and indicate that a collision hasoccurred. In a preferred configuration, the wall turns red. Similarly,if a knee “collides” with a bomb in the simulation, the bomb turns red.In an exemplary embodiment, an appearance of the object is altered inresponse to the collision. In exemplary configuration, the collisiontriggers a sound; the sound can be a directional sound indicating adirection of the collision with respect to the user or observer. Varioustypes of sounds can further provide information regarding the collision,such as, severity or the objects involved in the collision. For example,a user hitting the user's head on part of an aircraft can result in adifferent sound than a object colliding with the floor. In addition, thesimulation can alter its behavior based on detected collisions, byopening a door or panel, for example. In addition, this collisiondetection feature can be used to facilitate the grabbing of a virtualobject within the simulation, permitting the object to be moved orotherwise manipulated in the virtual environment. Collisions can alsooccur between avatars and between simulated objects, such as, between asimulated hand tool and simulated wall.

Embodiments of the present invention can also include, for example, animmersive observation environment as well. The observation environmentallows designers to observe and interact with the virtual realitysimulation including the avatars 56 (see FIG. 9), which can be driven inreal-time by motion capture data from the users. As illustrated in FIG.1, the observation environments can include the CAVE (Cave AutomaticVirtual Environment) 44, a reconfigurable, e.g., 8-foot-high by10-foot-wide by 10-foot-long room with displays on three walls and thefloor, where one or more observers view a common environment in animmersive way, including stereoscopic and full-scale images. Asunderstood by those skilled in the art, the CAVE 44 from Mechdynedisplays a resolution of 1280×1024 at 96 Hz. Therefore, designers andother observers can view in real-time detailed interaction of the userswithin the simulation. In an exemplary configuration, one wall of theCAVE 44 may be used to display all individual viewpoints with the othertwo walls and the floor of the CAVE 44 being used to display an overallview. In this configuration, an observer immersed in the simulation canview avatar 56 posture and the avatar view point simultaneously. Otheravailable data, such as temperature, distance measurements, time, etc.whether visible or not, may be also be displayed on the one wall of theCAVE 44, according to an exemplary configuration.

In an exemplary embodiment of the present invention, the software usedto display the 3D scene is based on OpenSceneGraph, a 3D graphics opensource toolkit. Much like the stand alone viewer for the head-mounteddisplay, the CAVE 44 can use the MiniViz software program. Theprojections of each screen, however, can be mapped using an ApplicationProgrammer's Interface (API) called CAVELib from Mechdyne. Also runningthe CAVE 44 are other programs, which can include: Vega, Vega Prime, andEnsight Gold, according to an embodiment of the present invention.

Embodiments of the present invention can also include head tracker 46(see FIG. 6) and wand hardware 48 (see FIG. 7), to be used in the CAVE44 to allow the observers to move around easily within the simulation.According to an exemplary embodiment of the present invention, theInterSense IS-900 uses a combination of inertial and ultrasonic sensorsto determine the positions of the head tracker and wand. The softwarerunning the head tracker and wand can include a program called Trackd5.5 from VRCO. As understood by those skilled in the art, the Trackdapplication takes information from the head tracker and wand and makesthat information available to either CAVELib or the ENVISION (D5)simulation. This tracked information is applied to correct theprojections on the walls and the floor of the CAVE 44.

Embodiments of the present invention can also include, for example, animmersive environment with the ability to switch between a simulationand a telepresence view. The telepresence view is available through theCAVE 44, a head mounted device 40, or desktop display. The content forthe telepresence view are gathered via a spherical camera 50 (see FIG.8A) at a remote location, such as, for example, the surface of anaircraft carrier. According to an embodiment of the present invention,the spherical camera 50 has a set of six digital cameras embedded intoone device that captures 75% of a sphere. The dynamic scene can bedisplayed in the head mounted display 40 or the CAVE 44, where the usercan be immersed in real-world video information to help validatesimulations. As understood by those skilled in the art, the Ladybug2 isa 4.7 MegaPixel video capture device using six 1024×768 CCDs. Accordingto an exemplary embodiment of the present invention, the Ladybug2 fromPoint Grey Research comes with a software development environmentLaybug2 SDK and a tool called LadybugCap that allows content to becaptured; this software produces spherical avi files for playback, asunderstood by those skilled in the art. In an exemplary embodiment,Ladybug3 can be used for better resolution but a lower frame rate.

As understood by those skilled in the art, the spherical camera 50produces data files in 1 GB increments, which may be 2 seconds or 2minutes. So, 30 seconds of video capture can turn into 15 files at 1 GBeach. These files require translation into a viewable format. Othersolutions use a video editor and paste the first viewable files togetherto form a single video file, in the process reducing the quality toproduce a smaller, second level video file. In contrast, embodiments ofthe present invention read the first level video files into buffers andprovide indexing, such as the first file, last, current, the file beforethe current and the file after the current. This allows the video groupwith many files to be played as if they were a single video file.

According to an embodiment of the present invention, real-world objectscan be scanned to create models for the virtual reality simulator. In anexemplary embodiment of the present invention, the Creatform Handy ScanEXAscan can be employed. In an exemplary embodiment of the presentinvention, a Leica HDS 3000 can be employed. The Leica HDS 3000 is alaser based device that scans equipment to quickly create high-fidelity3D models where they do not yet exist. As understood by those skilled inthe art, in practice, the true resolution of the scanner is ¼″ of pointaccuracy for unmodeled data and ⅛″ of point accuracy for data modeledbased on multiple points from a point cloud. According to an exemplaryembodiment of the present invention, the software used to capture thearea, set the resolution, and register the point clouds together isCyclone. Also used is PolyWorks to register the clouds and producepolygon geometry to be incorporated into the simulation. According to anexemplary embodiment of the present invention, the hardware supportingthe device can be a Dell Precision M70 Laptop, 1.86 GHz, 1 GB RAM, andNvidia Quadro FX Go 1400.

Embodiments of the present invention can include, for example, aportable motion capture system for capturing tasks in the field.According to an embodiment of the portable motion capture system, thesystem can include markers at predetermined locations, a motion capturesuit, a plurality of cameras installed on a tripod so that cameras cantrack the movements of a user wearing the motion capture suit, and acomputer or other storage medium to record the images from the camera.According to another embodiment of the portable motion capture systemfor capturing tasks in the field, the system can include markers atpredetermined locations, a motion capture suit, a plurality of camerasclamped on a rigid structure so that cameras can track the movements ofa user wearing the motion capture suit, and a computer or other storagemedium to record the images from the camera and to digitally recordlocations of the markers associated with the bodysuit, responsive to therecorded images from the plurality of cameras. In a trainingapplication, for example, motions of a remote trainer at a firstlocation can be tracked and captured for training of a trainee at asecond location, ever later or in real time. In addition, motions of oneor more users can be tracked and captured for task analysis, including,for example, ergonomic and efficiency analysis. In another embodiment,for example, a portable motion capture system can be utilized forreal-time design evaluation in the field and for design presentation ordemonstration, including, for example, a new design.

The embodiments of the present invention include a method of evaluatingan engineering design, as illustrated in FIG. 11. The method includessimulating a CAD design in a virtual reality environment (step 80) anddriving one or more avatars within the virtual reality simulation usingmotion capture data obtained from a user interacting with the virtualreality simulation (step 81). The method continues with displaying thevirtual reality simulation, including the interactions of the one ormore avatars, to multiple observers in a common, immersive environmentin real-time so as to evaluate the CAD design (step 82) to therebyverify that tasks associated with a product built according to the CADdesign can be performed by a predetermined range of user sizes.

The embodiments of the present invention include a method of validatinga simulation with real-world video using immersive technology, asillustrated in FIG. 12. The method includes capturing real-world videoby a spherical camera at a remote location (step 90) and rendering thevideo in a head mounted display (step 91). The method continues withcapturing the head rotation information of a user by a motion capturesystem (step 92) and controlling the pan, tilt, and zoom of the video bythe head rotation information of the user (step 93). The method alsoincludes switching, under user control, between displaying thereal-world video and the simulation (step 94).

The embodiments of the present invention also include a computer programproduct, stored on a tangible computer memory media, operable on acomputer, the computer program product comprising a set of instructionsthat, when executed by the computer, cause the computer to performvarious operations. The operations include, for example, receiving CADdata, generating video signals to simulate in virtual reality the designfrom the CAD data, providing for the tracking of multiple usersinteracting with each other and the simulation, providing for thetracking of objects interacting with the simulation, generating scaledavatars within the simulation, generating video signals for the commonimmersive environment, and receiving user input to select between video,graphics, or both together.

Embodiments can also include a computer program product, being stored inone or more tangible computer readable media and readable by a computerso that the computer program product operates to perform instructionsdescribed herein when read by the computer. The instructions includerecording at a first location full-body motion capture data for one ormore trainers performing one or more tasks by a portable motion capturesystem. The instructions include animating one or more avatars within avirtual reality simulation by a virtual reality simulator at a secondlocation, responsive to recorded motion capture data for the one or moretrainers at the first location so that each of the one or more trainerscorresponds to one of the one or more avatars. The instructions includedisplaying the virtual reality simulation, including the one or moreanimated avatars, as a three-dimensional image that appears to surroundone or more trainees to thereby define a common immersive environmentusing one or more head mounted displays so that the one or more traineescan analyze the one or more tasks performed. The instructions can alsoinclude obtaining motion capture data for one or more traineesinteracting with the virtual reality simulation through a motion capturesystem; animating one or more avatars within a virtual realitysimulation by a virtual reality simulator in real time, responsive tomotion capture data for the one or more trainees at the second location;detecting a collision between an avatar animated by a trainee and asimulated object in the virtual reality simulation by the virtualreality simulator; and altering a color of the simulated object in thevirtual reality simulation by the virtual reality simulator to providefeedback for the detected collision.

The embodiments of the present invention can also include a system fortraining at a remote location, for example, tasks associated withoperation or maintenance of an aircraft. Likewise, tasks can beassociated with the operation or maintenance of a design for anaircraft, a space system, a spacecraft, a ship, or a missile system.

Embodiments of the present invention further include a method ofsimulating a task. The method includes recording full-body motioncapture data for one or more users performing one or more tasks by aportable motion capture system. The method includes animating one ormore avatars within a virtual reality simulation by a virtual realitysimulator responsive to motion capture data for the one or more users sothat each of the one or more users corresponds to one of the one or moreavatars. The method includes displaying the virtual reality simulation,including the one or more animated avatars, as a three-dimensional imageusing one or more head mounted displays so that each of the one or morehead mounted displays can provide a different perspective of the virtualreality simulation.

The system includes a portable motion capture system 30, 42 at a firstlocation positioned to track the movements of one or more users, e.g.,trainers, and to record full-body motion capture data for one or moreusers, e.g., trainers, performing one or more tasks. The system caninclude a virtual reality simulator 58 being positioned to receive therecorded motion capture data from a first location and capable ofanimating one or more avatars 56 within a three-dimensional virtualreality simulation at a second different location, responsive torecorded motion capture data. The system can include an immersiveobservation system to display the virtual reality simulation, includingthe one or more animated avatars 56, as a three-dimensional image thatappears to surround one or more trainees to thereby define a commonimmersive environment 20 using one or more head mounted displays 40 sothat each of the one or more head mounted displays 40 can have adifferent perspective of the virtual reality simulation and so that theone or more trainees can analyze the one or more tasks performed.

It is important to note that while embodiments of the present inventionhave been described in the context of a fully functional system, thoseskilled in the art will appreciate that the mechanism of at leastportions of the present invention and/or aspects thereof are capable ofbeing distributed in the form of a computer readable medium ofinstructions in a variety of forms for execution on a processor,processors, or the like, and that the present invention applies equallyregardless of the particular type of signal bearing media used toactually carry out the distribution. Examples of computer readable mediainclude but are not limited to: nonvolatile, hard-coded type media suchas read only memories (ROMs), CD-ROMs, and DVD-ROMs, or erasable,electrically programmable read only memories (EEPROMs), recordable typemedia such as floppy disks, hard disk drives, CD-R/RWs, DVD-RAMs,DVD-R/RWs, DVD+R/RWs, flash drives, and other newer types of memories,and transmission type media such as digital and analog communicationlinks. For example, such media can include both operating instructionsand operations instructions related to the design and evaluation programproduct and the method steps, described above.

In the drawings and specification, there have been disclosed a typicalpreferred embodiment of the invention, and although specific terms areemployed, the terms are used in a descriptive sense only and not forpurposes of limitation. The invention has been described in considerabledetail with specific reference to these illustrated embodiments. It willbe apparent, however, that various modifications and changes can be madewithin the spirit and scope of the invention as described in theforegoing specification and as defined in the attached claims.

1. A system for providing a simulated environment and telepresence usingreal-world video, the system comprising: a spherical camera positionedto capture real-world video at a first location; one or morehead-mounted display devices at a second location, the one or morehead-mounted display devices being positioned to display the capturedreal-world video from the first location and also to display asimulation corresponding to the real-world video; one or more computersto support the one or more head-mounted displays; a motion capturesystem being positioned to capture head rotation information for one ormore users to thereby to control a pan, tilt, and zoom of the real-worldvideo responsive to the captured head rotation information for the oneor more users so that when a position of a user's head changes, theportion of the real-world video displayed in the head-mounted displaychanges accordingly; and a computer program product, stored on atangible computer readable memory media, operable on a computer, thecomputer program product comprising a set of instructions that, whenexecuted by the computer, cause the computer to perform the operationsof: receiving user input to select between displaying the real-worldvideo and the simulation in the head-mounted display.
 2. A system ofclaim 1, wherein the spherical camera positioned to capture real-worldvideo at the first location includes six digital cameras embedded into asingle device so that the single device can capture about 75% of asphere.
 3. A system of claim 1, wherein the operations further comprisereading the video files from the spherical camera into buffers andindexing the files so that a video group with many files can be playedas a single video file.
 4. A system of claim 1, wherein the operationsfurther comprise superimposing a portion of the simulation representinga field of view onto the portion of the real-world video rendered in thehead-mounted display.
 5. A system of claim 1, wherein the motion capturesystem includes up to 24 cameras mounted on a truss and, for each of theone or more users, gloves and a body suit with markers at predeterminedlocations on the suit.
 6. A system of claim 1, further comprising avirtual reality simulator being positioned to receive CAD design datafrom a CAD design and to create a three-dimensional virtual realitysimulation from the CAD design data so that the one or more head-mounteddisplay devices can display the simulation.
 7. The system of claim 6,wherein the CAD design data is for an aircraft.
 8. A system forproviding a simulated environment and telepresence using real-worldvideo, the system comprising: a spherical camera positioned to capturereal-world video at a first location; an immersive observation system ata second location, including one or more computers and a reconfigurableroom with displays on three walls and the floor, the displays beingpositioned to display the captured real-world video from the firstlocation and also to display a simulation corresponding to thereal-world video; a motion capture system being positioned to capturehead rotation information for one or more users to thereby to control apan, tilt, and zoom of the real-world video responsive to the capturedhead rotation information for the one or more users so that when aposition of a user's head changes, the portion of the real-world videodisplayed in the head-mounted display changes accordingly; and acomputer program product, stored on a tangible computer readable memorymedia, operable on a computer, the computer program product comprising aset of instructions that, when executed by the computer, cause thecomputer to perform the operations of: receiving user input to selectbetween displaying the real-world video and the simulation in thehead-mounted display.
 9. A system of claim 8, wherein the sphericalcamera positioned to capture real-world video at the first locationincludes six digital cameras embedded into a single device so that thesingle device can capture about 75% of a sphere.
 10. A system of claim8, wherein the operations further comprise reading the video files fromthe spherical camera into buffers and indexing the files so that a videogroup with many files can be played as a single video file.
 11. A systemof claim 8, wherein the operations further comprise superimposing aportion of the simulation representing a field of view onto the portionof the real-world video.
 12. A system of claim 8, further comprising avirtual reality simulator being positioned to receive CAD design datafrom a CAD design and to create a three-dimensional virtual realitysimulation from the CAD design data so that the one or more head-mounteddisplay devices can display the simulation.
 13. The system of claim 12,wherein the CAD design data is for an aircraft.
 14. A method ofdisplaying a simulation with real-world video using immersivetechnology, the method comprising: capturing real-world video by aspherical camera at a first location; rendering to a user at a secondlocation a portion of the real-world video in a head-mounted display,the portion of the real-world video representing a field of view, thehead-mounted display being positioned to display the real-world videoand to display a simulation related to the real-world video; capturinghead rotation information of the user by a motion capture system;controlling a pan, tilt, and zoom of the real-world video responsive tothe captured head rotation information of the user so that when aposition of the user's head changes, the portion of the real-world videorendered in the head-mounted display changes accordingly; and switching,under user control, between displaying the real-world video and thesimulation in the head-mounted display.
 15. A method of claim 14,wherein the step of rendering to a user at a second location a portionof the real-world video in a head-mounted display involves reading videofiles from the spherical camera into buffers and indexing the files sothat a video group with many files can be played as a single video file.16. A method of claim 14, further comprising superimposing a portion ofthe simulation representing a field of view onto the portion of thereal-world video rendered in the head-mounted display.
 17. A method ofclaim 14, further comprising comparing the simulation with thereal-world video to validate the simulation to thereby provide feedbackon the simulation.
 18. A method of claim 14, further comprising trainingthe user at the second location to perform one or more tasks at anenvironment substantially similar to the environment of the firstlocation.
 19. A method of claim 14, wherein the user is one user of aplurality of users at the second location, and wherein each user of theplurality of users is simultaneously rendered a user-controlled portionof the real-world video in a head-mounted display associated with saiduser so that each user can have a unique field of view.
 20. The methodof claim 19, wherein the motion capture system supports at least 6users, and wherein the head-mounted display provides to the user astereoscopic field of view.